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ABSTRACT 

The intent of this project was to improve the qualit 
of the high school experience for a selected group of "high risk" 
freshmen by providing them with services beyond those normally made 
available* At each of six high schools, the guidance counselors 
together with the counselors of each school's feeder schools, 
selected a group of 250 incoming students whose records indicated 
severe learning disabilities that would make them higher potential 
dropouts* In general^ these students who appeared economically as 
well as educationally disadvantaged, had to travel by bus to reach 
their schools which were situated in middle class economic settings* 
Among the additional services and modifications provided for these 
students were smaller classes, specially modified schedules, lighter 
course loads, special remedial instruction, intensive guidance and 
school-home liaison, and referrals to a school psychologist when 
needed* All six high schools were part of a simple project with an 
overall concern for support^ing, guiding and enriching the school 
experience of an atypical ^roup of students* Nevertheless, each 
school exercised a great deal of autonomy and operated a program 
unique to that school. In part these differences in program may have 
been directed by differences in the incoming student population. In 
part the differences in program may have been dicated by the balance 
or unbalance of strengths, values, and predilections among the 
project personnel involved.l (Author/JM) 
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Program Description 

The intent of this project was to improve the quality oC the high school experience for 
a selected group of 'Miigh risk" freshman by providing them with services beyond those 
normally made available. At each of six high schools, the guidance counselors together with 
the counselors of each school's feeder schools, selected a group of 250 incoming students 
whose records indicated ''severe learning disabilities that make them higher potential drop- 
outs.'' In general, these students who appeared ''economically as well as educationally 
disadvantaged," had to travel by bus to reach their schools which were situated in middle 
class economic settings. Among the additional services and modifications provided for these 
students were smaller classes, specially modified schedules, lighter course loads, special 
remedial instruction, intensive guidance and school-home liaison, and referrals to a school 
psychologist when needed. The staff designated to provide these services at each school 
included the following: 

1 coordinator (full-time teacher); I grade advisor (full-time teacher); I full-time school 
psychologist ( 10 hours per week); and I family assistant (25 hours per week). The 
average estimated cost per student tor these services was S240. 

All six high schools were part of a simple project with an overall concern for 
supporting, guiding and enriching the school experience of an atypical group of students. 
Nevertheless each school exercised a great deal of autonomy and operated a program unique 
to that school. In part these differences in program may have been dictated by differences in 
the incoming student population. For example, at School A the average reading score of 
PIANO students was 4.94, while at School B the average was 7.35. In part the differences in 
program may have been dictated by the balance or unbalances of strengths, values, and 
predilections among the project personnel involved. For example, while the reading program 
at one school stressed specific skills, the program at another school gave more emphasis to 
thematic understanding and appreciation of literature. Procedurally, the students at one 
school attended PIANO only classes, at another PIANO remedial classes only, and at 
another were completely integrated into the regular school program. Thus at some schools 
there was an attempt at getting the students to achieve a strong identification with the 
PIANO project and at another school most of the PIANO students were unaware that they 
were part of a special program. 

O 
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It should be noted in passing that these programmatic dilTerences between schools are 
not reflected in the evaluation objectives nor for that matter, WiS any systematic attempt 
made to compare the possible effects of these differences. 

Program Objectives 

The proposal for PIANO stated the following program objectives: 

1 . To facilitate and ease the adjustment of disadvantaged alienated youth to high 
school. 

2. To provide special supportive services to facilitate the student's awareness of his 
own potential and to help him develop that potential to the fullest extent. 

3. To provide special guidance services for educational and vocational planning 
during and after high school. 

4. To provide intensive remediation services to those students in the ninth year who 
need such services. 

5. To open up and maintain a contact with the student's home and parent to 
facilitate resolution of problems which may be preventing the student from 
achieving. 

6. To develop a climate in which the home and school can work together to create in 
the student a feeling of belonging to and identification with the school. 

7. To improve attendance patterns to facilitate learning. 

Evaluation Objectives 

Rouglily corresponding to these program objectives are the following objectives of 
evaluation which were prepared by the Bureau of Educational Research of the City of New 
York and appended to the PIANO proposal: 

1 . To determine whether at least 75% of the ninth grade students in each school in 
the program will improve by at least 8 school months in reading and arithmetic 
achievement from April 1972 to April 1973 on an appropriate standardized 
achievement test, 

2. To determine whether at least 75% of the ninth grade students in the program in 
each school will improve by at least two scale points in appropriate school 
behavior from initial to final rating by the grade advisor. 
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3. To determine whether at least 75% of the ninth grade students in the program in 
each school will improve by at least two scale points in motivation and positive 
attitude toward school from initial to final rating by the grade advisor. 

4. There will be a 30% increase in attendance of the ninth grade students in the 
program in each school. 

5. There will be a decrease of at least 35% in disciplinary infraction of the ninth 
grade students in the program. 

b. There will be an increase in home and parental contacts leading to a resolution of 

student referral problems. 
7. 60% of the ninth grade students in the program in each school will pass all of their 

subjects in January 1973: 75%: of the ninth grade students in the program in each 

school will pass all their subjects in June 1973. 
In the next section, data related to each objective will be descrilx*d and analyzed. 



Recommendations 

The recommendations in this section fall into two general categories, those concerned 
with evaluations of projects of this type, and those concerned with the design of future 
projects. 

Concerning evaluations 

1 . It is strongly recommended that the personnel of new programs that are to be 
evaluated should be directly and seriously consulted in the development of . 
evaluation objectives, criteria and measurement procedures. Not only might this result 
in better staff morale and attitude toward evaluation but it might also result in better 
evaluation designs. For example, had the project staff been consulted, they would have 
emphatically pointed out that in the original evaluation design it was completely 
unrealistic to base the evaluation of the changes in attitude and behavior of 250 
students on the judgment of a single grade advisor. It is unlikely thai a single person 
could have an adequate enough knowledge of such a large group of students to be able 
to rate them accurately and reliably. 
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2. Objectives should be supported by some rationale or hypotheses and should be 
based on past data when such data are available. For example, the expectation of 

the achievement of 8 months growth in as many months as measured by a standardized 
test normed on a national sample, may be unrealistic for a group of learners that has 
been gaining at a consistently slower rate over a period of many years. As previously 
discussed, objectives based on a historical regression approach may be more sensible in 
this situation. 

. Similarly, there probably exist extensive and relatively reliable attendance data 
for the students in experimental projects from which future trends, in terms of 
such factors as schools, seasons, and sex, could be predicted with relative accuracy. 
Changes in these trends due to some programmatic treatment could then be equated 
with ''effect size" in experimental research. Hypotheses regarding the amount of 
change expected could then be offered in terms of the standard deviation of past 
distributions of attendance data. For example, a large change in attendance could be an 
increase of a single standard deviation over last year's average for some group. A small 
change might be a quarter of a standard deviation and so on. The point being that the 
objective would be stated in terms related to past data, current recommended research 
procedures, and based on some hypotheses or rationale for expecting an effect of a 
specific size. 

3. Objectives should be sensitive to differences in goals and procedures at different 
project units. As discussed previously, different schools in the PIANO project operated 
fairly divergent programs and yet all were evaluated by the same criteria. Furthermore 
it is recommended that where there are differences among units, some systematic 
attempt be made to analyze the possible effects of these differences. 

4. The persons who will actually implement the project evaluation should have the 
opportunity to interact with the project personnel at the very early stages of the 
project. This early involvement may be important for the following reasons: 

a. there would be agreement on the precision of the data to be collected, times of 
test administrations and formats of records for recording and storing data, thus 
making the data collection procedures more efficient and less costly and allowing 
the data analysis procedures to be more powerful. 



ERIC 



b. the improvement in efficiency and "etTectiveness of the data collection and 
analysis procedures could lead to interim or formative judgments which could be 
fed back into the project in time to guide its procedures on the basis of its current 
effectiveness or lack of it (e.g. if it was realized that the remedial reading program 
at School A, or the remedial math program at School C or the attendance rate at 
School D was significantly different than those at other schools, the "causes'' of 
these differences might have been inferred and generalized to the programs at the 
other schools); 

c. the formative or guiding effect of the evaluation data might have initiated a 
feeling among the staff that evaluation procedures were an integral and helpful 
part of the project rather than merely a summary judgment of how well they had 
done their jobs; 

d. procedures for monitoring and describing the programs at each school, in 
particular their differences, could have been established to guide each unit in 
pursuing the uniqueness of its objectives. 

Summary 

In summary, we note that we cannot say, unequivocally, that the PIANO project 
accomplished any of the evaluations objectives set for it. In fact, this is the second consecu- 
tive year that evaluators have reached this judgment. Perhaps it is time to consider the 
alternative reasons for this state of affairs. 

The first alternative that comes to mind is that the evaluation objectives are unrealistic 
If this was simply an after the fact determination, this alternative might not merit seriousj 
consideratioii. However, there are certain aspects of the situation which do indicate that t us 
alternative be seriously considered. For example, the PIANO evaluators for the previous 
year clearly stated their judgment that the objectives last year, which were similar to this 
year's, were **unrealistic/' Either they "demanded too much growth of PIANO students" on 



they "did not tap the salient features of the programs." However, if anything, the objectives 
this year may have been even more demanding. In setting lorth an expectation (as in j 
objective one) that at least 75% of the students in PIANO will achieve at least 8 months 
growth during their 8 months in the Program before post-testing, an expectation is being set 
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forth that PIANO students will grow at a rate faster than the normal population. Based on 
the generally dismal record of the PIANO students' previous 8 school years this expectation 
does indeed seem unrealistic, in this regard predicted achievement based on individual 
historical regression seems a clear improvement, as previously discussed. 

Beyond this on at least two different occasions early in the project, the PIANO staff 
sent written communications to the central office expressing their specific dissatisfactions 
with the original evaluation design. However, neither the staff, nor their objections were 
consulted in developing the evaluation design. For these reasons, past data, statements of 
previous evaluators, and the early expressions of staff dissatisfaction, a reasonable argument 
can be presented that the project was not being evaluated by fair or appropriate criteria. 

Another alternative reason for the project's non-achievement may have to do with the 
relatively short period of time during wiuoh the project is expected to effect a significant 
change of behavior. For students with 8 school years of failure and 14 to 16 years of 
problematic existence, 8 school months may be a relatively short period of time to effect a 
change. Perhaps a more reasonable evaluation objective would consider a longer time in the 
project, i.e. two academic years, and a third and fourth year follow-up study. 

A final alternative reason for lack of project achievement must of course be that the 
project is reaching too few students with any degree of success. This may be so because the 
staff-to-student ratio is too high, because the resources are too little, because the procedures 
used are inadequate or some combination of all three. At any rate before such a project as; 
PIANO is recycled, serious consideration should be given to the adequacy of the program, in 
terms of its objectives and its target population. It is possible that some cherished 
hypothesis, e.g. that concernin^.^f>ii*.effect of a devoted and concerned counselor on an 
underachieving student, needs to be dropped or seriously modified. It is possible that some 
new thinking might prove helpful. In this regard there is one final recommendation to be 
made. 

In almost every case where an analysis of the difference between schools was made, 
that analysis indicated statistically significant differences. Unfortunately since the data in 
hand do not lend themselves to casual inferences, very little can be said regarding the 
reasons for these differences. And yet these differences may be the most interesting aspect 
of the data analysis. If some schools were operating programs which were significantly 
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different in terms of educational procedures and if these different procedures resulted in 
achievement which was significantly different statistically, then this is precisely the informa- 
tion tor which experimental research and development projects are looking. 

Unfortunately, neither in the design nor spirit of the PIANO project did inter-school 
comparisons play a significant role. Perhaps it is not too late to amend this situation. As a 
final recommendation then it is suggested that during the early fall of 1973, this evaluation 
team have some block of time (e.g. a week) to present the data it collected and its analysis 
of inter-school differences to the PIANO coordinators and counselors From all six schools. 
Perhaps together these people might be able to go back over the^previous year's procedures, 
extenuating circumstances and the like and arrive at some educated guesses regardhig the 
causes of the inter-school differences. This effort would be undertaken more in the spirit of 
research than evaluation and the guesses arrived at could be considered as hypotheses or 
possible guidelines to be investigated by future personnel. 

\ 
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ANALYSIS OF OBJECTIVES 



Objective One 

The first ol")jective, as stated in the original evahuUioii design, was: 

To determine whether at least 75% of the ninth grade students in each school in the 
program will improve by at least eight school months in reading and arithmetic achieve- 
ment from April, 1972, to April, 1973, on an appropriate standardized achievement test. 

Sampling scheme. The original design stated that achievement scores be collected and 
analyzed for all ninth grade PIANO students in each school. It was not possible to fulfill this 
specification. When two sets of tests scores are being compared, the comparison should be 
done only on students who participated in both test administrations. The range of number 
of students among the si.x schools who completed a particular initial and final achievement 
test varied from a high of 157 to a low of 37 which is approximately 63% and 15% of the 
students in the program at those respective corresponding schools. The sample sizes of avail- 
able students were modest so no additional sampling was done. Therefore, the sample con- 
sisted of all students at each school whose records showed that they had taken a particular 
test in 1972, and a second one in 1973. 

Sampling bias. The low percentage of participation in achievement test administrations 
suggests the operation of a self-selection process in taking tests. If so, the scores collected 
may not be representative of the population of ninth grade PIANO students. Perhaps the 
more conscientious, more highly motivated, more successful test takers are over represented 
in this sample. Unfortunately, no evidence is available to resolve this question. All that can 
be done is to alert the reader to the possibility of such bias. 

Test administration and instrumentation. The reliability of achievement test scores for 
these students, particularly the 1972 scores administered in the junior high schools, was 
nearly universally questioned by the PIANO personnel in the schools with whom we dis- 
cussed achievement test scores. Because of this problem several of the schools had readmin- 
istered achievement tests in the Fall of 1972. These later scores were supplied to us by the 
schools and were used in computing achievement gains. This resulted in time differences for 
the 1972 administration. Two schools (B and C) supplied us with reading scores collected in 
April, 1972; Schools E and F administered reading tests in October, 1972, and Schools A 
and D did not administer tests until November, 1972. Since more complete and (sup- 
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posedly) more reliable data were available from these Sater tests they were used in making 

i 

the comparison to be reported. i 

Little hard evidence is available on the question of achievemeiU test score reliability. 
One indication of unreliability comes from examining the change scores between the 1972 
and 1973 administrations of the reading achieven}ent test,s. Of 570 students from the six 
PIANO schools, tlie change scores of 62 (11%) indicated a (Jecrccise\of more than eight 
months over the school year. We are not questioning the reliability of this achievement test 
under certain conditions and for certain populations of students. However, the reliability of 
the data represented here is questionable. 

School A preferred a different type of reading achievement test to the Metropolitan 
Achievement Test which was the instrument used in all the other schools. They admin- 
istered the Gates-MacGinitie Reading Survey in November, 1972, and April, 1973. The 
scores. for this test are grade equivalent scores (like the MAT). The correlation on 82 
students between the 1973 scores on Gates-MacGinitie and the 1973 MAT was ,703. The 
means were .5.80 and 5.33 and the standard deviations wer>* L59 and 1.47, respectively. The 
distribution of Gates-MacGinitie scores were very similar tc the MAT, they were highly cor- 
related, and the Gates scores were more complete and more available. They were therefore 
used as the reading achievement scores for this school. 

The MAT in mathematies was apparently not administered on a city-wide basis in the 
Spring of 1973 but was to be administered by the PIANO schools to the PIANO students. 
Only three of the six schools administered mathematics tests. One of these was not compa- 
rable to tlic MAT in either content or in form of the scores, and was therefore not included. 
Both of the remaining schools administered the MAT test in Fall, 1972 and again in Spring, 
rv^73. 

DifferefU'Cs among schools. This evaluation was not charged with comparing the six 
schools in the PIANO program. Because we suspected that differences among schools would 
contribute significantly to the error variance, we designed many of the analyses to identify 
and remove this component of variability. The purpose in so doing was primarily to increase 
the precision of the analysis. In all cases, the differences among schools were statistically 
signitlcant, supporting our expectatioij. 
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Because a comparison oT schools was not requested we did not intend to collect 
inlbrmation to dilTerentiate among schools. However, the observations we made at the 
schools, the genera! descriptions olTered by the PIANO statT, and the data collected all made 
us realize that the populations of students attending the six schools differed and the pro- 
grams implemented at the schools differed. 

Without identifying particular schools it may be of interest to point out some of the 
differences observed among the sclv^ols. One reason for incorporating this information in 
the report is that we believe tl ./fferences raised questions aboUv the suitability of using 
the same criteria for evaluating all schools. This point will be elaborated in the section on 
recommendations. 

The reading data (1972 scores) provided one basis for noting that the student popula-- 
tions differed. A one-way analysis of variance on 1972 reading scores indicates that scores 
do differ significantly among schools. The F-ratio for the effect of schools was 29.8 with 5 
and 564 degrees of freedom. The tabled value of F (o^ = .05) is approximately 2.25. Dif- 
ferences among the schools accounted for 21% of the total reading score variance. The mean 
1972 reading scores for each school are presented in Table 1. 

Analysis of reading achievetnent data. The stated objective for reading achievement was 
that 75% of students gain at least eight months in reading during the ten month school year. 
Because the 1973 reading achievement tests were administered in April, students were in the 
PIANO program only eight months when this final measurement was taken. The objective 
therefore requires that 75% of students gain at least approximately the normal rate for this 
test: one month for each month in school. In the light of these students' previous reading 
histories is this a reasonable objective? 

Table I presents information on reading scores for the six schools in the PIANO pro- 
gram. For example, the average 1972 reading score for a student enrolled in School A was 
4.94. This average increased to 5.78 in 1973, resulting in a gain of .83 years or 8.3 months. 
The standard deviations of these distributions are also included in thj.* table. Finaly the per- 
centage of students in each school gaining at least eight months is given. The range of 
percentages is 3 1 .6 to 46.9, None of the schools are Jiear the objeclive of 75%. 

Analysis of adjusted data. It was previously stated that some of the 1972 tests were 
administered as late as November. The gains reported for these schools were computed over 
an interval of only five months. This does not permit a reasonable comparison among these 
schools. 
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TABLE 1 

Mean and Standard Deviations on 1972 and 1973 
Standardized Reading Achievement Tests, Gain Scores, 
and Percentage of Students Gaining Eight Months or 
more between the Test Administrations 

Schools 





A 
M 


D 
D 








1 


1972 Test 














Mean 


4.94 


7.35 


7.04 


7.32 


5.83 


5.20 


Standard Deviation 


.96 


2.15 


2.32 


2.25 


1.73 


1.33 


1973 Test 














Mean 


5.78 


8.07 


7.48 


7.50 


6.27 


5.70 


Standard Deviation 


1.53 


2.29 


2.50 


2.24 


1.80 


1.87 


Gain Score 


.84 


.72 


.44 


.18 


.44 


.50 


Percentage of Students 


44.3 


41.4 


34.1 


31.6 


46.9 


44.3 



gaining eight months 
or more 

An estimate of gain for a standard interval cf eight months can be computed by 
dividing the actual gain score of each student by the number of months between the respec- 
tive test administrations and multiplying this ratio by eight. There is a question whether the 
most appropriate interval should be eight months or ten months. As the evaluation design 
was set up (to utilize test administrations already scheduled and regularized for the city), 
the measure of gain is confounded by combining two months of pre-PIANO education and 
eight months of PIANO education. If the expectation is that rate of growth in reading will 
be significantly higher in PIANO than it was prior to PIANO then it would he more 
straight-forward, simpler, and more conservative to consider only the eight months in 
PIANO. Therefore an eight month interval was used in computing adjusted scores. 

School A had the largest actual average gain of all the schools. These scores were 
adjusted to sec if adjusted scores for the school with the largest gain would meet the reading 
objectives. The mean increased from .84 to 1.11 meaning that approximately 50% of the 
students exceeded a gain of eleven months. If 50% exceed a gain of eleven months, did 75% 
exceed a gain of eight months? The actual gains scores for students in School A were 
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multiplied by 8/6 and the pereeiitage of adjusted scores equaling or exceeding eight months 
was computed. The number of scores equaling eight months or more increased from 35 to 
48 increasing the percentage who met the objective from 44.3 to 60.8. The adjusted scores 
for School A stil) fall considerably short of meeting the stated objective of 75%. Since 
School A had the largest gain none of the schools would meet the criteria even if adjusted 
scores were computed. 

TABLE 2 

Summary of Information on Mathematics 
Achievement Test Data 

Schools 



Sample Size 66 37 

Testing Dates 9/72, 5/73 11/72, 3/73 

Interval between Tests in Months 8 6 

1972 Test 

Mean 5.80 4.96 

1973 Test 

Mean 6.69 5.02 

Percentage of Students 56. 1 1 8.9 

exceeding eight month gain 

Predicted 1973 scores 6.34 5.25 

t'test between predicted 3.87 -1.94 

and actual 1973 scores 

Analysis of mathematics achicvefncnt data. As previously mentioned mathematics 
achievement data w«':s available from only two schools. Pertinent information about these 
data are presented in Table 2. The gains for neither school is near the objective for mathe- 
matics which is the same as that for reading -75 percent of students gaining more than eight 
months. 
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Historical regression analysis of achievement in reading. Another nietliod of analyzing 
gain scores is called the ''historical" regression method. This technique uses the 1972 
reading score as a measure of reading ability that is assumed to have increased at a constant 
rate over a student's entire length of time in school. Using this assumption, an average 
reading gain per school month is computed for each student. This average gain is then 
multiplied by the number of months in the PIANO program prior to the I ^)73 ics(. In this 
way an estimate of achievement gain for the length of time in PIANO is computed based on 
a student's previous gain in (supposedly) conventional programs. If the PIANO program is 
effective the actual gains achieved in this program should be higher than that extrapolated 
from previous experience in conventional programs. A correlated l-test is used to compare 
predicted versus actual 1 973 scores. A statistically significant difference (ifi favor of (lie actual 
score) is accepted as evidence that the PIANO program was an effective one. 

The data for performing an historical regression analysis were computed and are disj^layed 
in Table 3. The 1973 actual score is higher than the predicted seore for all schools except 
Schools C and 0. However the amount of the difference must be sufficiently large to conclude 
that there is a real difference between test score distributions for the population of PIANO 
students. If the computed value of the t statistics is greater than the tabled value of t for the 
appropriate degrees of freedom and for the level of significance specified, then we can con- 
clude that Mie ditterences between the populations are large enough to be considered real ones. 
The tabled value for t (one-tail test) with level of significance of .05 ajid 80 degrees of freedom 
is 1.665. 

Students in School A increased in reading ability an amount greater than that predicted 
by the historical regression technique. School F falls just a hair below the cut-off value for t so 
it wiii be considered to have also increased by a significant amount. 

Although it was not anticipated that a school might fail to gain as much as predicted, 
such was the case for School D (for a two-tailed test with = .05, and 100 degrees of 
freedom the tabled value of t is 1.984). Students in the remaining three schools did about as 
well as predicted by the historical regression method. 

hi case there is any doubt in the reader's mind the 1973 reading scores are higher than the 
1972 scores. Even School D produced a t of 1.696 on a correlated t-test. The tabled t for 100 
degrees of freedom and ^ = .05 is 1 .660 (one-tail test). 
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TABLE 3 

Summary of Information on Historical 
Regression Method of Analyzing Reading 
Achievement Data 



Schools 





A 


B 


C 


D 


E 


F 


1973 Actual Score 


5.78 


8.07 


7.48 


7.50 


6.27 


5.70 


1973 Predicted Score 


5.23 


8.00 


7.66 


7.70 


6.18 


5.46 


Actual Gain 


.84 


.72 


.44 


.18 


.44 


.50 


Predicted Gain 


.29 


.65 


.62 


.38 


.35 


.26 


t- values 


5.834 


.697 


-1.330 


-1.816 


.418 


1.66 


Sample Size 


79 


157 


91 


114 


49 


80 



Historical regression analysis of achievement in mathematics. The same procedure 
described above was applied to the mathematics data. The results are reproduced in Table 2. 
The computed value of t for School C (3.87) is larger than the appropriate tabled value 
( 1 .67 1 ). The average gain of 8.9 months is significantly higlier than the predicted gain of 5.4 
months. F'ailure to gain the predicted amount in School E is almost significant. 

Discussion of historical regression technique. The historical regression method is superior 
to a fixed criterion in at least one way. This method is sensitive to individual past history. The 
criterion on which a student is judged is one established by his own past performance. If a 
student, during the experimental period, does better than a measure predicted by his past 
history, the experimental treatment is judged to be eflective. 

The greatest problem with this method is to obtain a reliable estimate of past 
achievement gains. The use of one score, and one that is suspected as being unreliable, to char- 
acterize the average growth in achievement over a long time span is not a satisfactory proce- 
dure. More data points are needed to develop a fair and reliable estimate of growth. 

Tlie assumption of a linear growth rate for students such as these in the PIANO program 
might also be questioned. An equally likely hypothesis might be that these students exhibit a 
negatively accelerating growth curve. An investigation of growth curves for a sample of 
students from the population over a period of time is needed to validate the shape of the 
growth curve appropriate for use in studies like this one. 
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Summary, Only one school (A) approached the stated objective for reading achievement, 
when gains were adjusted to an eight month interval, but did not meet it. Two schools (A and F) 
exhibited reading gains greater than that predicted by the historical regression technique. Of 
the two schools that provided mathematical achievement scores, neither achieved the stated 
objective but one school (C) achieved higher scores than that predicted by historical regres- 
sion. The schools that did achieve higher scores in mathematics and reading are not the same 
schools. The different results achieved by analyses raise questions about the setting of criteria 
for evaluation that will be discussed in the recommendations section. 

It is interesting to point out that the school (A) with the lowest entering scores of all six 
schools had the largest actual gain in reading of all schools. If the PIANO program were the 
san^e at all schools this llnding would be very unexpected. This result suggests that something 
different was happening at School A. From what we know of the reading programs, the one at 
School A was a more structured, remedial program than was true of programs at other schools. 
Perhaps the goals of the reading program at School A were more nearly matched to the reading 
achievement tests than were the programs at other schools. 

For both the reading and mathematics areas, one school in each area, and not the same 
school, failed to maintain the previous growth rates of students in these respective areas. This 
finding further supports the previously expressed conviction that the programs at the PIANO 
schools are not the same programs. Much more might be learned about educational efforts 
similar to the PIANO program by comparing and analyzing the differences implemented by 
different schools in the program and the effects associated with these differences by ignoring 
these differences. 
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Objective Two 

The original objective was: 

To cleterniiiie whether at least 75% of the ninth grade students in the program in 
each school will improve by at least two scale points in appropriate school behavior 
tYom initial to tlnal rating by the grade advisor. 

The design proposed that a ten item, five-point rating scale be developed by the evaluation 
agency.and that it be used by the grade advisor to evaluate all PIANO students in October 
and again in May. ''Appropriate school behavior" was not detlned by the design. Hxaniples of 
items suggested in the design were: 

1 . Whispers or talks in class 

2. Talks back to teacher 

3. Withdrawn from groups 

Since the original design could not be followed, because of the lateness in contracting for the 
evaluation, the pre-post test design was changed to one of comparing PIANO students with a 
group of non-PIANO students who were as similar to PIANO students as possible. Attending 
to or participating in classroom activities was judged to be the essential aspect of student 
behavior sought by the original design. Therefore it was selected as the behavior to be 
measured via a modified Jackson-Hudgins Observation Schedule (Labadene, H., 1967) with a 
PIANO and non-PIANO group in each school. 

Procedure for using Jackson-Hudgins OI)servation Schedule, The procedure was for the 
observer to ''sweep" the room, looking at each student in turn, and immediately recording the 
students' state of attention on a matrix observation schedule prepared in advance. Four classi- 
tlcations were possible: a plus was recorded if the student was attentive; a minus was recorded 
if the student was clearly inattentive; a question mark indicated that the observer was 
uncertain what the student's state of attention was: and a zero signified that attention was not 
observable. The observer continued to sweep die students in the classroom for an entire class 
period. The number of completed sweeps varied from 1 5 to 50 depending on the number of 
students in the classroom. The sum of the tallies entered for each of the four attention cate- 
gories were totaled for each student in the classroom to obtain an attention profile. 

Inter-observer reliability, defined as percentages of agreement between two or more 

ERIC 



17 

judges, ranged from 86 percent to 99 percent tor a series of observations made by the 
originators of this observation schedule. 

Classroofn observation. At least two classrooms were observed in each of the six 
schools. The PIANO staff selected one PIANO class and attempted to match it with an 
equivalent non-PIANO class. The observations were completed within a week and a half to 
minimize the effect of uncontrolled influence on behavior. 

Analysis of data. Since the number of question marks and zeroes was quite small they 
were dropped from the analysis and attention was focused on the frequencies of pluses and 
minuses or states of attention and inattention, respectively. As can he seen in Table 4 the 
overall behavior of the non-PIANO students was significantly different and in this case 
significantly more attentive than the PIANO students. The differences in frequencies of 
states of attention and inattention amongst these two groups of students with an N = 187 
yielded a chi-square of 65.5, well beyond chance expectations. 

TABLE 4 

Chi-square Analysis of Attentiveness of PIANO 
and non-PIANO Students 

Groups 

PIANO Non-PIAIMO 

Attentive 1601 1914 

Non-Attentive 573 377 

However, an examination of the results, school by school, indicates that in three 
schools the PIANO students were significantly more attentive than non-PIANO students 
while in only two schools was the reverse situation found. This schooi-by-school look also 
indicates that when PIANO sl?-;(lents were inattentive they were very inattentive, to a much 
greater degree than the inattentive non-PIANO students. This extreme ditTerence in degree 
in the two schools accounts for the overall difference between PIANO and non-PIANO 
students, A further analysis of the conditions in School A which ditTered in the extreme and 
accounted tor the greatest part of the overall ditTerence indicates an unusual condition. 
Specifically, the room temperature at the time of the observations was above 95'\ a 
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condition which would stiniuiate inattention on the pyrt of most high school students. How- 
ever, the non-PIANO group was highly motivated that day due to the fact they received and 
discussed their final examination of the term. Allowing for this atypical set of circumstantvc 
and dropping School A from the analysis, PIANO students were not significantly more 
inattentive than non-PIANO students. The chi-square value was .097. To be significant 
chi-square must exceed 3.84 with one degree of freedom and Oi = .05. However, the weak- 
ness in these procedures is apparent. 1 his observation schedule is sensitive to the nature of 
the lesson being observed as well as to the style of the teacher and any atypical conditions 
which might prevail. Therefore, in order for valid conclusions to be inferred from data 
collected with this schedule it should be used with a much larger, more representative 
sample of classroom lessons for both PIANO and non-PIANO groups. 
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Objective Three 

The original objective was: 

To determine whether at least 75% of the ninth grade students in the program in 
each school will improve by at least two scale points in motivation and positive 
attitude toward school from initial to final rating by the grade advisor. 

The original design proposed that the evaluation agency prepare a ten item, five-point 
ratuig scale. The grade advisor was to rate every ninth grade PIANO student in the school in 
October and again in May on "motivation and positive attitude toward school." Four 
sample items suggested in the design were: 

1. Having self confidence 

2. Having good work habits and study habits 

3. Wanting to learn and improve himself 

4. Volunteering to do things in class 

The proposed five-point multiple choice response ranged from "Always" to "Never." The 
original pre-test post-test design could not be implemented. A one-time comparison between 
PIANO and non-PIANO students was proposed as an alternative. 

Attitude toward school. Two different attitude questionnaires were selected to be 
administered to samples of PIANO and non-PIANO students in each of the six schools. 

The Student Opinion Poll (SOP) developed by Getzel and Jackson (1962) was selected 
to measure attitude toward school. The forty nine items on the test measure attitude toward 
teachers, the curriculunu the student body and classroom procedures. The student checks 
the one response of three or four alternative responses to each question that best matches 
his opinion of the school he is attending. One response is designated as the most satisfactory 
response. Each time this response is selected a score of one is given. 

A sample item from the questionnaire is: 

13. Most of the subjects taught in this school are 

a. very interesting 

b. above average in interest 

c. below average in interest 

d. dull and uninteresting 
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A copy of the entire questionnaire is included in Appendix A. The questionnaire was orig- 
inally prepared for use with sixth graders. One reason for selecting this instrument was the 
expectation that this questionnaire would be readable by students in the PIANO program. 
Reliability of the instrument From previous administrations with sixth grade students varied 
tVom .85 to .89. 

One change was made in the questionnaire. An item that read '*as preparation for high 
school, the program of this school is .... " was changed to replace high school with college. 

Ohiective for attitude toward school. It was very difficult, with the amount of informa- 
tion available to us about both classes of PIANO students and classes of non-PlANO 
students that would be judged to be similar to PIANO classes students, to establish expecta- 
tions about differences in attitudes toward school between these two groups. No directional 
hypothesis was therefore stated for attitudes toward school. Instead data was sought to 
determine whether the attitudes toward school as measured by the SOP would be signif- 
icantly different for PIANO students as compared to non-PIANO students. 

The analysis of data should be a t-test or appropriate Analysis of Variance to compare 
the PIANO and non-PIANO students. The results of the analysis will be reported below. The 
questionnaire was to be adminiskTcd in May or June 1973 io selected classes of PIANO and 
non-PIANO students. It was understood that the non-PIANO classes closely equivalent to 
PIANO classes do not exist in most of the schools but that classes most similar to tiie 
selected PIANO classes would be used as reference groups. A convenient sample of classes, 
although not desirable, was accepted as necessary to minimize the practical problems in 
obtaining clas.ses for collecting this information. 

Attitude toward counseling services. The staff at several of the schools seemed to feel 
that one of the most effective parts of the PIANO program was the special counseling 
services provided for PIANO students. We selected, as a second attitude instrument, a 
questionnaire designed by Form (1955) to assess attitudes concerning counseling services 
available at school. The original questionnaire is a twenty two item scale with five response alterna- 
tives from "Strongly Agree'' to ''Strongly Disagree." The student responds by circling the response 
that indicates ho\w he feels about the services. A sample item is: 

It is a complete waste of time to go to the counseling otTice. 

Strongly Agree Agree ? Disagree Strongly Disagree 
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The split-lniir reliability of the instrument on a sample of 544 college students was .94, One 
item was dropped from the instrument and the wording of several items was simplified. 
Specific names were also changed to use terminology more commonly associated with 
counseling facilities at high schools. A copy of the questionnaire is included in Appendix B. 
Each iu^m on the Counseling Attitude Scale was scored from one to five depending on 
which of the five responses was checked by the respondent. The possible total score range of 
the instrument was 21 to 105. 

Ohjectivc for uttimde lowunl counseling service. The objective for counseling attitude 
was to demonstrate that the PIANO students had more favorable attiUidcs toward counsel- 
ing facilities at school than did the noii-PIANO students. The method of data analysis was a 
t-test or an Analysis of Variance. Administration of the questionnaire and sampling procedures 
were to be the same as specified for attitudes toward school. 

Saniplif}^ scheme. The PIANO staff at each school was requested lo select one PIANO 
class and one non-PlANO class, that was as similar to the PIANO class as possible, to respond to 
the questionnaires. In a few cases, when class sizes were small, additional classes were used 
to increase the sample si/e. The number of students responding to the questionnaires was 
,>9^) on the Student Opinion Poll and 385 on the Counseling Attitude Scale. A schedule for 
administering questionnai es was arranged with the PIANO staff at each school. One class 
period was suftlcient for students to respond to both questionnaires. 

Analysis ofattiiudes toward schools. Responses to the Student Opinion Poll were 
collected from 196 PIANO and 203 non-PIANO students. The questionnaires were scored 
and analyzed with a two-factor analysis of variance. The two factors were groups (PIANO 
and non-PIANO) and schools. The analysis of variance table is presented in Table 5. 

There was no difference in responses between the two groups. The percent of variance 
accounted for by group membership was very small (. I %). There was a significant difference 
among schools. Schools accounted for 38''/ of the total variance. The mean attitude scores 
for each school, in the order School A to F, are: 19.9, 20.2, 16.3, 19.1,20.0, 18.2. 

A nalysis of attitudes toward counseling services. The Counseling Attitude Scale was 
responded to by 193 PIANO and 192 non-PIANO students. The same analysis procedure 
was applied lo these data as lo the Student Opinion Poll. The analysis of variance table for 
these data is presented in Table 6. 



TABLE 5 



Analysis of Variance for Student Opion Poll 



Groups (PIANO - Non-PIANO) 

Schools 

Error 



Sum of 
Squares 

10.325 
776.354 
18.405.392 



Degrees of 
Freedom 

1 

5 

392 



Mean 
Square 



F ratio 



10.325 <1 
155.271 3.31 
46.95 



Total 



19.181.746 



Tabled F: ( a = .05, df = 5,120) = 2.29 



398 



TABLE 6 



Analysis of Variance for Counseling Attitude Scale 



Effect 

Groups (PIANO - Non-PIANO) 

Schools 

Error 

Total 



Sum of 
Squares 

507.871 
917.448 
61,407.216 

62,832.535 



Degrees of 
Freedom 

1 
5 

378 



Mean 
Square 

507.871 
183.489 
162 



F ratio 

3.1 
1.1 



384 



Tabled F: (a=.05, df - 1,120) -3.92 



The elTect of neither of the two factors uiuilyzed were lurge enough to reject tlie null 
hypothesis of no clilTerence among levels of the factors. It is concluded that tliere were no 
differences in attitudes toward counseling services for either of the two groups or among the 
six schools. The objective stated for attitudes about counseling services was not attained. 

Summary, PIANO and non-PIANO students did not differ in attitude toward school in 
general or in attitude toward the counseling services. The expected higher attitude toward 
counseling service by PIANO students was not supported by the data. There were dif- 
ferences toward school among the schools but such differences were not found in attitude 
toward counseling service. 

O 
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Objective Four 

The original evaluation design proposed: 

To determine whether there was a 309? increase in attendance of the ninth grade students 
in the PIANO program in each school. The sample was to be the entire population. Attend- 
ance w'^^ to have been measured and compared between an initial period of October and 
November 1972 and a final period of April and May 1973. A percentage comparison 
between the two periods was requested. 

Sampling scheme. It was proposed, and accepted, that rather than sample attendance at 
an initial and final period, samples would be taken throughout the school year to determine 
if attendance patterns were subject to seasonal variations. For example, attendance may 
increase gradually over the year but drop suddenly in late April or May because of weather 
conditions. The sampling scheme originally proposed would not detect such variations. By 
monitoring one week randomly selected from each month these kinds of patterns can be 
identified and the amount of variation determined. 

It was also proposed, and accepted, that data not be collected for the entire population 
but that a random sample of students be selected; that data be collected on this sample, and 
that these data be analyzed to test this objective. We decided to eliminate the tlrst 
(September) and last (June) months of the school year from the population because they 
may contJ*»in unusual effects associated with the opening and closing of school. January was 
also dropped because of the large number of days in the month when school was closed. l\n 
the remaining seven months, all weeks not including holidays were identified and one week 
each month was randomly selected as the sampling period for tiie month. The periods 
selected were: October (2-6), November (13-17), December (4-8), February (5-9), March 
(26-30), April (2-6), and May (14-18). 

A sample of 25 male and 25 female students were selected from each ninth grade 
PIANO school population. The population was defined as students who were in the PIANO 
program throughout the year and who were freshmen for the first time in September 1972. 
The equal sampling of male and feniale students was done in order to look for comparisons 
between sexes. The list of students who comprised the population was prepared and nth 
student was selected: n being an appropriate number to produce approximately 25 students 
from one pass through the list. In most schools additional names had to be sampled from 



:4 

the list bocmiso records of one kiiul or another could not be found or a selected student 
luul recently moved or l)een transferred from the program. 

Annlvsis of att{.'iulance data, llie attendance data was recorded and analyzed as 
number of absences during one five-day week. Table 7 displays the average number of 
absences per sampling period over the year by sex and school. For example, the average 
number of absences per sampling period over all seven sampling periods for males at School 
(* was .7.> ilays. Ivmalesat this school had a nearly equal rate of absenteeism, .71 days per 
sampling period. 

TABLE 7 

Average Number of Absences by Sex and School 

Average for 



Sex 









Schools 






All Schools 




A 


B 


C 


D 


E 


F 




Male 


1.71 


.78 


.73 


.55 


.99 


1.81 


1.00 


Female 


.94 


.96 


.71 


.44 


.93 


1.26 


.87 



A repealetl measures four way analysis of variance was computed on the absence data. 

The four fjctors were the seven sampling periods, six schools, two sexes and students. 
(Sliidcnls appear as faclt^rs in repealed measures designs. The variaHon among students is 
iiearly always significant if it is evaluated. The effect of this factor will be ignored in this 
analysis and also in those in the following objective. ) Repeated measures were over the levels 
of sampliniz periods. Details of the analysis, which is somewliat complex, are described in 

Appendix (*. Tiie result of the analysis tiemonstrates tlifferenees among sampling periods, 
among the schools, and a sex by school interaction effect. The objective of this analysis was 
to delermme whether :ittendance increased during the year. The variations among sampling 
periods indicate lunv attendance (or rather absences) changed over the yean 

1 hero was a slalistically signiHcant difference among the sampling periods. Tlie mean 
number of ai^sences per student per sampling period, beginning with October and ending 
with April, was 0.7 | , | | .20. 0.<SrK 0.95, and 1 .20. It appears that absences start out low 
each semester and increa.se as the semester progresses. Tiie same pattern occurs in both the 
I all and Spring semesters. ( The reader should recall that January was not sampled so the 
O.Sh datum is for Ivbruary. ) 
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Since the design called Tor it, a comparison was made between the October-November 
average attendance and the April-May average attendance. The absence data was converted 
to mean number oT presences per student per samphng period by subtracting absences from 
total numl^er ofdays in the sampling period (5). The average number of presences was 4.2^), 
3/)l, 3.cS(), 4. 14, 4. 12, 4.05, 3. SO. The average attendance during October-November is 4. 10 
and during April-May is 3.^)25. The change is a decrease, not an increase. The percentage is 
4.27. Although a statistical test was not supplied to this comparison this dilTerence is so 
small it is not likely that the clilTernce is significant and it should not be interpreted as a 
meaningful change in attendance. The appropriate conclusion is that attendance was no 
different at the end of the year than it was at the beginning. The goal of the objective was, 
of course, not met. 

The analysis showed that there were differences in number of absences among the six 
schools. In order. School A through School f\ the average number of absences per sampling 
period over the year were 1.33, .87, .72, .50, .96, 1.53. School F had three times as many 
absences per sampling period as did School I). 

Although the difference between sexes was not quite large enough to be significant at 
the .05 level of confidence, the sex by school interaction was significant. The data of this 
comparison is reproduced in Table 7. In Schools A and F the boys were absent more than 
the girls In Schools C, I) and li the number of absences for both sexes was very similar. In 
School B the girls were absent more than the boys. 

SuniNUwy. There was a change in attendance among the sampling periods throughout 
; he year. The change was not of the nature stated as the objective for these data. It did not 
increase from the beginning to the end of the year. Attendance appears to have been highest 
at the beginnmg of each semester of the year and to have decreased as the semester pro- 
gressed. The goal of this objective was not met. 

There were differences in attendance among the six schools and there was a significant 
interaction between schools and sex. In two schools (A and F) boys had more absences than 
girls. In School B girls were iibseiU more than boys: in the remaining schools the dif- 
ferences between sexes were small. 
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Objective Five 

Vhc original cvukijiion design established a goal of a decrease of at least 35 percent in 
disciplinary infractions of ninth grade students in the program over the school year. All 
ninth grade PIANO students in each school were to be used. The number and type of dis- 
ciplinary referrals 5br the period October and November 1972 were to be compared with 
April and May I ^^73, A percentage between periods was to be done. 

This objective was changed to sample the student population and to expand the 
sampling periods of disciplinary data from two two-month periods to four two-month 
periods: October-November, December-January, February-March, and April-May. The same 
sample of students used in Objective Four was used for this objective. 

In all schools, except one. disciplinary matters were the province of the deans. In one 
school disciplinary infractions of PIANO students were referred to {he PIANO counselor. 
There were wide variations from school to school in the type and extent of records kept. 
The only classification scheme that developed out of discussions with several of the school 
personnel was to identify as major infractions referrals that resulted in suspension and to 
classify all others as minor infractions. The records for each student in the sample were 
located in the Dean's files. Kvery referral was examined and classified as major or minor and 
entered in the appropriate sampling period according to the data of the referral. 

Analysis oj disciplinary ififractions. The number of major infractions was very small. A 
total of seven major infractions were reported among the six schools. Six of these were 
reported by School F and involved four infractions by girls (two by the same girl) and two 
by boys. I-our of the seven infractions occurred in the April-May reporting period. There 
was not a sufllcient number of major infractions to do any further analysis of these data. 

A repeated measures analysis of variance (similar to the one used in Objective Four) 
was perlornied on the minor infraction data. The four factors of the analysis were the four 
sampling periods (with, repeated measures over this factor), the six schools, sex, and students. 
The detailed struculre of the analysis is presented in Appendix D. The result of the analysis 
was to demonstrate dilTerences among the sampling periods, among the schools, between 
sexes, and an interaction between sampling periods and schools. 

The average number of infractions changed significantly over the four sampling periods 
from 0.H2 to 0.^)2 to O.^^l and to 0.89. The December-January sampling period was much 
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lower than the other three. However, there was not a drop from the first period to the last 
period. An increase of 8.5% was computed between the beginning and ending sampling 
periods but the dilTerence is so small it is not likely that it is significant. With the exception 
of the December-January period the number of infractions remained fairly constant. In any 
case the objective of a 35% decrease was not achieved. 

There were other differences in the data. The number of infractions among the schools 
differed significantly. For schools A to F the average number of infractions per student per 
sampling period were .34, 1.20, 1 .48, .52 .43 and .90. More than four times aji many infrac- 
tions were reported at School C than at School A. It may be of interest to note that School 
A, where discipline for PIANO students was administered by the PIANO counselor, reported 
the lowest number of infractions. Not enough is known about the disciplinary procedures 
from school to school to relate these differences to specific practices. 

There was a difference between sexes in the number of minor infractions. The average 
number reported for boys over the year was .97 and for girls .66. 

There was also a significant interaction between schools and sampling periods. Such a 
finding indicates that the variation in minor infractions across sampling periods is not the 
same for each school. While infractions increased over the year for ':ome schools, it 
decreased for others and was nearly constant for still others. These data are also difficult to 
interpret without more detailed information about each school's disciplinary situation. 

- Summary. The goal of objective five was not met. The only change in the average 
number of minor disciplinary infractions over the year was a decrease for the Decem- 
ber-January sampling period. Males were recorded as having committed more minoi infrac- 
tions than females. Significaru differences among schools and an interaction effect between 
schools and sampling periods were not interpreted for lack of information about disciplinary 
practices at each .school. The number of major infractions (suspensions) were too few to 
analyze. 
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Objective Six 

The original evaluation objective stated that there would be an increase in home and 
parental contacts leading to a resolution of student referral problems. 

By the lime this evaluation began, the bulk of the family-home contacts had already 
been completed. Unfortunately, the evaluators did not have any opportunity to inlluence 
the keeping of records relating to these contacts. Consequently, although in most cases there 
appears to have been a very large and useful amount of energy expended by the PIANO staff, 
in particular by the Family Assistance Counselors, the records of these activities are dif- 
feaMit from school to school, generally informal or anecdotal and on the whole extremely 
difficult to interpret in terms of the orginally stated objective. For example, in many cases it 
was not possible to make a determination from the available records whether a problem was 
successfully resolved or not. To make this interpretation would have required lengthy con- 
sultations with the Family Assistance Counselors at each school. In most cases either she or 
the evakiation staff felt that time spent hi such pursuit would be too costly in terms of 
responsibility to other phases of the project. As a result we can only report that many dis- 
cussions with PIANO staff, school administrators. Family Assistance personnel as well as 
examination of available records, all indicate that these activities were considerable in effort 
and achievement. On this basis there seems little doubt that such activities should be con- 
tinued but that consistently uniform and specific record keeping should be encouraged. In 
contrast, we might note in passing, the general conversations revealed no such enthusiasm 
concerning the efforts and achievements of some of the school psychologists connected with 
the project. Nor, for that matter, was a specific objective related to this project explicitly 
designated for evaluation. 
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Objective Seven 

The orginal design stated in its objective Tor class grades that 60% of all ninth grade 
PIANO students in each school pass all their subjects in January 1973 and that this increase 
to 75% passing all subjects in June 1973. 

The only change in this objective was to sample the student population rather than to 
collect data tor the entire population. The sample of 50 students from each school on which 
attendance and disciplinary data was collected was used to collect grades. 

The raw data of this analysis was the proportion of courses passed. This form was used 
for two reasons, it was relatively easy to classify recorded marks as pass or fail but it was 
ditTicult to arrive at numerical scores for each course. Secondly, this form was convenient to 
use for reporting percentages passing all classes. 

Analysis of grades. A similar four factor repeated measures analysis of variance was 
computed on the grades data, A detailed description of the analysis is available in Appendix 
E. The result of the analysis is that there were significant effects due to sampling periods 
(January and June), schools, sex, and a sex by sampling period interaction. The sex by 
school interaction was not evaluated because an appropriate denominator term was unavail- 
able for the analysis. 

There was a difference in grades between the two sampling periods. The average pro- 
portion of courses passed decreased from January to June. The mean January proportion 
was .66; the June mean was .57. About 50%^ of the students passed two-thirds of their 
classes in January but only three-fifths of them in June. Table 8 shows the percentage who 
passed all courses in January and June at each school. Obviously the goal of this objective 
was not met, 

TABLE 8 

Percentage of students in each school who passed 
all courses in January and June 

A B C D E F Total 

January, 1973 2 36 47 46 36 22 31.3 

June, 1973 2 23 42 42 26 4 24.0 
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The other results of the iiiuilysis indicates thut the proportion of courses pussecl varied 
signincantly among schools, hi the order School A to School F the average proportions 
passed lor the year were .35, .74, .75, .77, .53, .55. Approximately 507f of the students in 
each or these schools passed at least the proportion of classes listed ahove. These per- 
centages are based on the assumption that the mean and median do not dilTer substantially. 
The assumption was not tested. 

The clilTerence ir^ grades between sexes was in favor of females who passed an average 
of .66 courses while males passed .57 of their courses. Finaly there was a sex by marking 
period interaction. The data of this interaction is presented in Table Although both sexes 
decreased in proportion of courses passed from Jaiiuary to June the size of the decrease was 
much larger for males than tor teniales. 

TABLE 9 

Average proportion of courses passed by 
Sex and Marking Period 

Males Females 

January .64 .69 

June .50 .63 

Sionniary. There was a change in grades between January and June but it was a 
decrease rather than the expected increase. I{ven in January the percentage of students who 
passed all their courses was 31 .3 instead of the expected 60. Additional differences were 
noted between sexes, with females passing more courses than males. The change in the pro- 
portion of courses passed from January to June differed tor sexes. The males decreased 
more than the temales did. There were also diflerences in grades among schools. Studeiits in 
Schools B, C and D, on the average, passed twice as many courses as students in School A. It 
is interesting to note in passing that School A is the school that had the greatest increase in 
reading of all schools. 



RELATIONSHIPS AMONG ATTENDANCE, GRADES, 
AND DISCIPLINARY INFRACTIONS 



By collecting atteiukince, grades, and number of disciplinary inrnjctions on each of the 
50 students sampled from eacli school, we are able to indicate how these three variables 
related to each other. Combining all schools and sexes, the correlation over the total sample 
was: 

Correlation between Grades and Absences -.652 
Ct)rrelation between Grades and Disciplinary Infractions -.217 
Correlation between Absences and Disciplinary Infractions -.103 

I*or the size of this sample (570) a correlation must exceed .! .09 to be signitlcant at the .05 
level of confidence. Thus all three of these correlations are significantly larger than zero, 
although the relation between grades and number of absences is much larger than the others. 

A high negative correlation between grades and absences indicates that a student' with a 
large number of absences is likely to have passed a low proportion of courses. Stated in 
another way, a student who was present most of the time is likely to have passed a high 
proportion of courses. No causal relation should be inferred from these data. High attend- 
ance is not to be interpreted as the course of higher grades or vice-versu. A modest negative 
correlation such as that between grades and disciplinary infractions also indicates a pattern 
of higher number of infractions being paired with lower proportion-of courses passed. How- 
ever the consistency of this relation is not as strong as with a high correlation. The proba- 
bility of the relation holding is less than is the case for a high correlation. 

CompiJtnig these correlations separately by sex shows very little difference between 
absences and grades ( females, ~.h5, males, -.66) but sexes vary in the relation between 
grades and number of disciplinary infractions. The correlation is higher for females (-.262) 
than tor males ( -.1 70). There is no correlation between absences and number of discipli- 
nary infractions for males (.022) although for females there is a small positive correlation 
(.239). The correlations by school and sex are reproduced in Table 10. Within a given set of 

variables, the correlations for females vary more among schools than is true for males. As 

j 

before, it is difficult to interpret these differences without more knowledge about inter- 
school differences. 
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TABLE 10 



Correlations among Absences, Disciplinary Infractions, 
and Course Grades by Sex and School. 



School 


Absences and Course Grades 


Absences and Disciplinary 


Course Grades 










Infractions 


and Absences 




Males 


Females 


Males 


Females 


Males 


Females 


B 


-.697 


-.895 


.249 


.427 


-.502 


-.614 


D 


-.531 


-.799 


.123 


.308 


-.299 


-.435 


C 


-.745 


-.755 


.123 


.590 


-.544 


-.613 


A 


-.574 


-.659 


-.099 


.025 


-.298 


-.216 


E 


-.620 


-.600 


-.029 


-.098 


-.010 


-.236 


F 


-.763 


-.506 


.010 


.240 


-.394 


-.331 



o 
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RECOMMENDATIONS 

The recommendations in this section fall into two general categories, those concerned 
with evaUuitions of projects of this type, and those concerned with the design of future 
projects. 

Concerning evaluations 

1. It is strongly recommended that the personnel of new programs that are to he 
evahiated should be directly and seriously consulted in the development of evalua- 
tion objectives, criteria, and measurement procedures. Not only might this result in 
better staff morale and attitude toward evaluation but it might also result in better 
evaluation designs. For example, had the project staff been consulted, they would have 
emphatically pointed out that in the original evaluation design it was completely un- 
realistic to base the evaluation of the changes in attitude and behavior of 250 students 
on the judgment of a single grade advisor. It is unlikely that a single person could have 
an adecjuate enough knowledge of such a large group of students to be able to rate 
them accurately and reliably. 

2. Objectives should he supported by some rationale or hypotheses and should be 
based on past data, when such data are availal^jle. For example, the expectation of 

the achievement of H months growth in as many months as measured by a standardized 
test normed on a national sample, may be unrealistic for a group of learners that has 
been gaining at a consistently slower rate over a period of many years. As previously 
discussed, objectives based on a historical regression approach may be more sensible in 
this situation. 

Similarly, there probably exist extensive and relatively reliable attendance data 
for the students in experimental projects from which future trends, in terms of such 
factors as schools, seasons, and sex, could be predicted with relative accuracy. Changes 
in these trends due to some programmatic treatment could then be equated with 
''effect size'' in experimental research. Hypotheses regarding the amount of change 
expected could then be offered in terms of the standard deviation of past distributions 
of attendance data. For example, a large change in attendance could be an increase of a 
single standard deviation over last year's average for some group. A small change might 



be a quarter of a standard deviation and so on. The point being that the objective 
would be stated in terms related to past data, current recommended research proce- 
dures, and based on some hypotheses or rationale for expecting an effect of a specific 
size. 

3. Objectives should be sensitive to differences in goals and procedures at different 
project units. As discussed previously, different schools in the PIANO project operated 
fairly divergent programs and yet all were evaluated by the same criteria. Furthermore, 
it is recommended that where there are differences among units, some systematic 
attempt be made to analyze the possible effects of these differences. 

4. The persons who will actually implement the project evaluation should have the 
opportunity to interact with the project personnel at the very (larly stages of the 
project. This early involvement may be important for the following reasons: 

a. there would be agreement on the precision of the data to be collected, times 
of test administrations, and formats of records for recordmg and storing data, 
thus making the data collection procedures more efficient and less costly and 
allowing the data analysis procedures to be more powerful. 

b. the improvement in efficiency and effectiveness of the data collection and 
analysis procedures could lead to interim of formative judgments which 
could be fed back into the project in time to guide its procedures on the 
basis of its current effectiveness or lack of it (e.g. if it was realized that the 
remedial reading program at School A, or the remedial math prog^-am at 
School C\ or the attendance rate at School D was significantly different 
from those at otiier schools, the "causes" of these differences might have 
been inferred and generalized to the programs at the other schools): 

c. the formative or guiding effect of the evaluation data might have initiated a 
feeling among the staff that evaluation procedures were an integral and help- 
ful part of the project rather than merely a summary judgment of how well 
they had done their jobs; 

d. procedures for monitoring and describing the programs at each school, in 
particular their differences, could have been established to guide each unit in 
pursuing the uniqueness of its objectives. 
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STUDENT OPINION POLL 

This is not a test. The answer to each question is a matter of opinion. Your true opinion, 
whatever it is, is the right answer. You will be asked a lot of questions about the school in 
which you are now studying. Wherever the words "school,*" ''teacher," and "student" appear, 
they refer to ihis schooK the teachers you have had while studying hert\ and your classmates 
in ////.V school. 

Here is an example: 

Circle the letter to the left of the answer you choose. 
0. In general I study 

a) too little 

b) too much 

c) about the right amount 

If your answer is "a) too little/' circle the 'a' in the first answer, like this: 
@ too little 

b) too much 

c) about the right amount 

If your answer is ''b) too much/' circle the 'b' in the second answer, like this: 
a) too little 
(b) too much 
c) about the right amount 

If you have any questions, raise your hand and you will be helped. 

Project for Increased Achievement and a New Outlook 
Directors: Donald Payne, Robert Ruderman 
New York University 

Center for Kducational Research and Field Services 
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1 . This school hstens to parents' opinions 

a. too much 

b. just enough 

c. too httle 

2. The number of courses given in this school is 

a. too many 

b. lust about right 

c. not enough 

3. Although teachers differ in this school, most are 

a. very good 

b. good 

c. fair 

d. poor 

4. In some schools the princioal sees and talks with the students often, while in other schools 
he rarely sees them, in this school the principal sees and talks with students 

a. too often 

b. just about the right amount 

c. too little 

5. The chance to say or do something in class without being called upon by the teacher is 



b. too much | 



a. too little 

b. too mucli 

c. about right ; 

i 

6. The things that I am asked to study are of 

a. great interest to me 

b. average interest to me 

c. of little interest to me 

d. of no interest to me 

7. Getting to know other kids in this school is 

a. easier than usual 

b. about the same as other schools 

c. more difficult than usual 

8. As pieparation for college, the program of this school is 

a. too tough 

b. about right 

c. too easy 

9. The class material from year (o year 

a. repeats itself too much; you learn the same material over and over 

b. repeats itself just enough to make you feel what was learned 

c. is so new that the things learned in the last grade do not help much in this one 
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10. In this school the teachers' interest in the students' school work is 

a. too great 

b. just about right 

c. not great enough 

1 1. When students in this school get bad grades, their classmates usually 

a. leel sorrier for them than they should 

b. admire them more than they should 

c. show the right amount of concern 

1 2. Students in this school are 

a. too smart - it is difficult to keep up with them 

b. just smart enough - we are all about the same 

c. not smart enough - they are so slow I get bored 

13. Most of the subjects taught in this school are 

a. very interesting 

b. above average in interest 

c. below average in interest 

d. dull and uninteresting 

14. The teachers' interest in what the students do outside of school is 

a. too great 

b. about right 

c. too small 

1 5. The student who shows a sense of humor in class is usually 

a. ndmired by the teacher more than he should be 

b. punished by the teacher more than he should be 

c. given about the right amount of attention 

16. When teachers "go too fast," students do not know what is going on. In this school, most 
teachers teach 

a. too slowly 

b. about right 

c. too fast 

1 7. Students who are good in sports are respected by classmates 

a. more than they should be 

b. less than they shoukl be 

c. neither more nor less than they should be 

1 8. The practice of competing against each other or of working together in this school 

a. leans too much toward comp*?tition 

b. leans too much toward working together 

c. is well balanced 
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19. On the whole, the things we study in this school 

a. are about right 

b. should be changed a little 

c. should be completely changed 

20. The teachers I have had in this school seem to know their subject matter 

a. very well 

b. quiiEe well 

c. fairly well 

d. not as well as they should 

21. Students may work either by themselves or in groups. In this school we work in groups 

a. too often 

b. just enough 

c. too little 

22. Students get along in this school 

a. very well 

b. about average 

c. not too well 

d. very badly 

23. The amount of **school spirit" at this school is 

a. more than enough 

b. about right 

c. not enough 

24. On the whole the school pays attention to the things you learn from books 

a. too much 

b. just enough 

c. not enough 

25. Teachers in this school seem to be 

a. almost always fair 

b. generally fair 

c. occasionally unfair 

d. often unfair 

26. The things we do in class are planned 

a. so badly that it is hard to get things done 

b. so well that we get things done 

c. so completely that we hardly ever get to do what we want 

27. Our seats in class 

a. change too much; we can never be sure where we will sit and who will sit next to us 

b. change about tho right number of times 

c. never change; we stay in the same place all year 
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28. The students who receive good grades are 

a. liked more than they should be by their classmates 

b. disliked more than they should be by their classmates 

c. neither hked nor disliked more than they should be 

29. hi this school the teachers' interest in the students' school work is 

a. just about right 

b. not great enough 

c. too great 

30. in my opinion, student interest in social affairs, such as clubs, scouts, and the **Y" is 

a. too great 

b. about right 

c. too little 

3 1 . hi general the subjects taught are 

a. too easy 

b. about right in difficulty 

c. too difficult 

32. When students need special attention, teachers in this school are 

a. always ready to help 

b. generally ready to help 

c. ready to help if given special notice 

d. ready to help only in extreme cases 

33. The ability of the teachers in this school to present new material seems to be 

a. very good 

b. good 

c. average 

d. poor 

34. In general, students in this school take their studies 

a. too seriously 

b. not seriously enough 

c. just about right 

35. In this school teachers seem to teach 

a. too many things that are not useful to us now 

b. too many things that are useful to us now but not later 

c. both things that are useful now and can be useful later 

36. When it comes to grading students, teachers in this school are generally 

a. loo ^nough" 

b. just ''tough" enough 

c. not ''tough" enough 
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37. The student who acts differently in this school is likely to find that most students 

a. dislike him for being different 

b. do not care whether or not he is different 

c. like him lor being different 

38. in niy opinion, students in this school pay attention to their looks and clothes 

a. too much 

b. about right 

c. too little 

30. In general, teachers in this school are 

a. very friendly 

b. somewhat friendly 

c. somewhat unfriendly 

d. very unfriendly 

40. In general, I feel the grades I received in this school were 

a. always what I deserved 

b. generally what 1 deserved 

c. sometimes what I did not deserve 

d. frequently what I did not deserve 

41 . Teaching aids such as films, radio, and the like are used 

a. more than they should be 

b. as much as they should be 

c. less than ihey should be 

42. Memory work and the learning of important facts are 

a. stressed too much 

b. used about right 

c. not stressed enough 

43. In some classes the teacher is completely in control and the students have little to say about 
the way things are run. In other classes the students seem to be boss and the teacher con- 
tributes little to the control of the class. In general, teachers in this school seem to take 

a. . lOo much control 

b. about the right amount of control 

c. too litiie control 

44. Some schools hire persons in addition to teachers to help students with special problems. 
In my opinion, this type of service \n this school is 

a. more than enough - it is often forced upon us 

b. enough to help us with our problems 

c. not enough to help us with our problems 
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45. When a new-comer enters this school, chances are that other students will 

a. welcome him 

b. ignore him 

c. dislike him 

46. Homework assignments in this school usually 

a. help us to understand 

b. have little to do with what we learn in class 

c. are just "busy work" 

47. In general, teachers in this school pay 

a. too much attention to individual kids and not enough to the class as a whole 

b. not enough attention to individual kids and too much to the class as a whole 

c. about the right attention to individual kids and to the class as a whole 

48. In general, my feelings toward school are 

a. very favorable — I like it as it is 

b. somewhat unfavorable — I would like a few changes 

c. somewhat unfavorable — I would like many changes 

d. very unfavorable - I frequently feel that school is pretty much a waste of 
time 

49. In this school the teachers' interest in the students' school work is 

a. not great enough 

b. too great 

c. just about right 
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APPENDIX B 
Counseling Attitude Scale 
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COUNSELING ATTITUDE SCALE 



We are interested in how you feel about the PIANO counsehng services available at your 
school. Your responses to the statements below will indicate your feelings. Read each 
statement carefully. Decide how you feel about it. Circle the response that best matches 
your feeling. Please respond to every statement whether or not you have had direct experience 
with the counseling office. 



Example 

I enjoy going to the counseling office. 
SA A ? D SD 

If you strongly agree, circle SA 
If you agree, circle A 

If you are undecided or uncertain, circle ? 

If you disagree, circle D 

If you strongly disagree, circle SD 



If you have any questions at this iime, please raise your hand. 
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1 . I think the counseling facilities are an important part of the school. 

: SA A ? D SD 

2. 1 feel the counseling office is unable to solve any kind of problem. 

SA A ? D SD 

3. Guidance given to troubled students by counselors is helpful to tiieni. 

SA A ? D SD 

4. The efforts made by the couaseling staff to help students are not practical. 

SA A ? D SD 

5. I believe the counseling office is u.seful in helping students with their problems. 

SA A ? D SD 

6. Talking with counselors at school reduces ten.sion, if nothing el.se 

SA A ? D SD 

7. It is a complete waste of time to go to the counseling office. 

SA A ? D SD 

8. I feel the counseling available at school is helpful to students who need it. 

SA A ? D SD 

9. It seems to me the counseling office is an important part of the school. 

SA A ? D SD 

10. I feel that '/ can not trust anyone at the couaseling office to help me. 

SA A ? D SD 
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11. I regard the counseling office as a place that is serving students with problems of 
adjustment. 

SA A ? D SD 

1 2. I believe the counseling office does not adequately interpret test results. 

SA A ? D SD 

13. The counseling office is a poor excuse for a place where students can take their problems: 

SA A ? D SD 

14. I believe the counseling office is a good means of helping students with their problems. 

SA A ? D SD 

1 5. The counseling office is of no direct help to students. They only tell you what you 
already know. 

SA A ? D SD 

16. I think more students should take advantage of the counseling service available at 
school. 

SA A ? D SD 

17. I believe the tests used by the counseling office are valuable. 

SA A ? D SD 

18. I believe the counseling office is not interested in students or their problems. 

SA A ? D SD 

19. 1 recommend the counseling service at school to all who need help. 

SA A ? D SD 
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20. I feel the school counseling office does enough good work to make it worthwhile. 

SA A J. ? D SD 

21. There is a complete lack of organization at the counseling office. You always get 
the run around. 

SA A ? D SD 
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Repeated Measures Analysis of 
Variance on Attendance Data 
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The three analyses computed on attendance, disciplinary infractions, and grades were 
very similar. A general model will be developed here for the anrJysis of attendance data. In 
the subsequent appendices the same model will be applied to different sets of data and if 
variations, are present they will be explained. 

All the analyses are four-way repeated measures designs with the repeated measures 
occuring over one factor; the. sampling periods or marking periods. The four factors are 
sampling periods (A), schools (B), se.x fC), and students (D). Students or subjects are usually 
treated as one factor in repeated measures designs. Students will be considered to be nested 
under two other factors; schools and sex. The analysis is shown in tabular form in Table I. 

The model for this design is: 

Yijkl = M + a I + /3k + Ti ^ ^i(jk) kl 
+ ayjr+ /37kj +aXii(jk) +a/37jki 

Table 2 presents the variance components for the analysis of variance on the attend- 
ance data, ft should be noted that there is no provision for an error term in the model 
because there is only one observation for each cell in the designs. Although the error term is 
the usual denominator when computing F ratios for an analysis of variance, this is not 
usually the case for repeated measures on mixed model designs, as will be seen. Several inter- 
actions are not included in the model. Interaction effects cannot be computed when one 
factor is nested under others. Therefore there is noi3X,aX, or no three-way interactions 
including these terms fai3X,a7X) or no four-way interaction. 

Cornfield and Tukey ( 1956) published a general algorithm for determining the ex- 
pected mean squares for different effects in experimental designs. The expected mean 
squares are used to decide what denominators are appropriate for each F ratio to be com- 
puted. The matrix generated by applying this algorithm is presented in Table 3. Note that 
the effects included are only those specified in the model. 

The variables Dp, Dq, Dp, and Dj^, are given values of either 0 or 1 depending on 
whether the respective factor is considered to be a random or fixed factor. Fixed factors are 
assigned a value of zero, random factors are one. For the attendance analysis, schools and 
sex are fixed ( Dq = Dp = 0); sampling periods and students are considered to be random (Dp 
= Dp = 1 ). These values are entered into the table to replace the respective variables. Dq and 
Dp should be set to one in those rows of the table where this effect appears as a nested 
factor: namely in the rows including the factor represented by X . 
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TABLE 1 

Schematic Outline of Experimentai Design (Attendance Data) 



School 
A 



School 
B 



School 



C Male 



Female 



r Male 



Mai*?. 



Female 



r S, 



^2b 



-'as 



r Si 



L s. 



25 



Female 



S 



'2 5 



r Si 



-"25 



r Si 



L s 



25 



Sampling Periods 



^Sample data for attendance was the number of absences during each of the seven 
five-day sampling periods. 



TABLE 2 

Analysis of Variance Table of Attendance Data 



Degrees of 

Effects Sum of Squares Freedom Mean Square 



Among Students 


2275.26108 


299 




Schools (B) 


259.58276 


5 


51.91655 


Sex (C) 


25.63406 


1 


25.63046 


B X C 


56.57523 


5 


11.31504 


Students (S) 


1933.47263 


288 


6.71345 


Within Students 


2062.19743 


1800 




Sampling Periods (A) 


62.18475 


6 


10.36413 


AxB 


36.64380 


30 


1.22146 


AxC 


6.20952 


6 


1.03492 


A X B X C 


34.02475 


30 


1.13416 


AxS 


1923.13461 


1728 


1.11293 


TOTAL 


4337.45703 


2099 
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TABLE 3 

Matrix for Determining Expected Mean Squares 



Effect 


1 


k 




i 


oc\ 


P 


q 


r 


n 


Pk 


p 


Dq 


r 


n 




n 


n 


Dr 


n 


^i (jk) 


P 


Dq 


Dr 


Dn 




Dp 


Dq 


r 


n 


ayjl 


Dp 


q 


Dr 


n 




P 


Dq 


Dr 


n 


a^il (jk) 


Dp 


Dq 


Dr 


Dn 


a/^Tjkl 


Dp 


Dq 


Dr 


n 



Table 3 corrected for use with the attendance data appears as Table 4. 



TABL E 4 

Matrix for Determining Expected Mean Squares 
for Attendanrfi Data 



Effect 


1 


k 


j 


i 


1. ai 


1 


q 


r 


n 


2. dk 


P 


0 


r 


n 


3. 7i 


P 


q 


0 


n 


4. ^i(jk) 


P 


1 


1 


1 


5. 0!/3kl 


1 


0 


r 


n 


6. ayji 


1 


q 


0 


n 




P 


0 


0 


n 


8. aXji (kj) 


1 


1 


1 


1 


9. a/?7jkl 


1 


0 


0 


n 



53 



The expected mean square for an effect is a weighted sum of every row in the table that 
includes the subscript (or subscripts) of the effect being evaluated. The weight of each 
component is the product of all entries in the respective row excluding the entries in the 
column (or columns) of the subscript (or subscripts) of the effect being evaluated. For 
example, the mean square for the effect of A must include components from rows 8, 5, 
and I in Table 4 since these effects include the subscript which is associated with a. 
The termaj37 is weighted by the product of every entry in row ^) except that in column 
'T': it is 0. Row 8 becomes simply with a weight of one; row 6, 0; row 5. 0; row K a 
with a weight of qrn. 

Adding these components together produces: 

Continuing this process for the remaining effects produce: 
li(MS«) = 5^+m5^^ + p5^ + prn 5^ 
H (MSr) = 5-. + qn 5- + p 5- + pqii 5- 

l-:(MSs) = 5^X + p5^ 
. H(MSBc) = "6^p^.5:^ + pn52^ 

To test each effect an P ratio of expected mean squares is formed such that the 
denominator contains all terms in the numerator, except the term representing the effect 
being evaluated. For example, the numerator of the F ratio for tlie main effect of A 
or sampling periods is 5^^^ + qrn 5^ . The last component represents the effect due to 
sampling periods. The denomintor should be simply 5^^^ . The correct term for the 
denominator is the interaction ierm between A and S. 

Main Effect A: F = = 

E(MSaS> s^x 

Main Effect B: (no appropriate denominator found) 
Main Effect C: (no appropriate denominator found) 

E(MSs) 5^> + p5r^ 



Main Effect S: F = 



erJc 
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Interactions AB: F 



AC: F 



BC: (no appropriate denominator found) 

The interaction of AS and AFC cannot be evaluated because of the lack of an error term 
in the design. 

Although simple denominators are not available to evaluate the effect of B, C, and BC, 
quasi-F ratios can be used to make these tests (Winer, 1968). Several expected mean squares 
are combined to produce the necessary terms for a denominator. For example,, the denominator 
for "B" requires the components: 

b\ +rn62^ + p62 

This can be obtained by the combination 

E (MSab) + E (MSs) - E (MSas) 

Therefore 

E (MSb) 

E (MSab) + E (MSs) -E (MSas) 

8\ + rn 52^+ p 52 
E (MSc) 

Main Effect C: F = 

E (MSac) + E (MSs) - E (MSas) 

E (MSbc) 

Interaction BC: F = 

E(MSabc) + E(MSas) 

The results of the computations for each of the evaluated effects are presented in Table 5. 



E(MSab)=^X^M^ 
E (MSas) 52j^ 

E(MSAC) = 52^ + qn52^ 
E (MSas) 52j^ 



Main Effect B: F = 



Also 
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TABLE 5 

F Ratios for Testing 
Significance of Effects on Attendance Data 

Degrees Value Approximate 
of ofF Tabled 

Effect F Ratio Freedom Ratio F Value (a =.05) 

Sampling 



Period (A) 


MS (A) / MS (AS) 


6/1728 


9.31* 


2.10 


Schools IB) 


MS (B) / MS lAB) + MS (S) - MS (AS) 


5/228 


7.61* 


2.27 


Sex (C) 


MS (C) / MS (AC) + MS (S) - MS (AS) 


1/197 


3.86 


3.92 


Students (S) 


MS (S) / MS (AS) 


228/1728 


6.03* 


1.35 


Ax B 


MS (AB) / MS (AS) 


30/1728 


i.io 


1.46 


Ax C 


MS (AC) / MS (AS) 


6/1728 


.93 


2.10 


B X C 


MS (BC) / MS (ABC) + MS (AS) 


5/980 


5.04* 


2.23 



*Significant at better than the specified .05 levei 



S 



ERIC 



APPENDIX D 



Repeated Measures Analysis of 
Variance on Disciplinary 
Infractions 
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This analysis is very similar to the one tor attendance data. Factor A of the analysis becomes 
sampling periods for disciplinary data which was four periods instead of seven. Otherwise the 
analysis is the same. Table 1 presents the component of variance and Table 2 the F ratios needed 
to test the significance of each effect. 

TABLE 1 

Analysis of Variance Table for 
Disciplinary Infractions Data 

Effects Sunfi of Squares Degrees of Freedom Mean Square 



Among Students 2004.68205 299 



Schools (B) 


214.58665 


5 


42.91733 


Sex (C) 


28.82999 


1 


28.8299 


B X C 


1729.91542 


5 


6.27000 


Students (S) 


1729.91542 


288 


6.00665 


Within Students 


1097.44307 - 


900 




Sampling Period (A) 


14.12667 


3 


4.70889 


Ax B 


36.47333 


15 


2.43155 


Ax C 


1.01667 


3 


0.33889 


A X B X C 


35.96333 


15 


2.39756 


AxS 


1009.86407 


864 


1.16882 


TOTAL 


3102.12612 


1199 





o 
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TABLE 2 

F Ratios for Testing 
Significance of Effects on Disciplinary Data 

Degrees Value Approximate 
of ofF Tabled 

Effect F Ratio Freedom Ratio F Value {a = .05) 



Sampling 
Periods (A) 


MS (A) / MS (AS) 


3/864 


4.03* 


2.65 


Schools (B) 


MS (B) / MS (AB) + MS (S) - MS (AS) 


5/102 


5.90* 


2.3 


Sex (C) 


MS (C) / MS (AC) + MS (S) - MS (AS) 


1/164 


5.57* 


3.9 


Students (S) 


MS (S) / MS (AS) . 


288/864 


5.14* 


1.2 


Ax B 


MS (AB) / MS (AS) 


15/864 


2.08* 


1.68 


A X C 


MS (AC) / MS (AS) 


3/864 


.29 


2.62 


Bx C 


MS (BC) / MS (ABC) + MS (AS) 


5/34 


1.76 


2.53 



•Significant at better than the specified .05 level. 
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APPENDIX E 

Repeated Measures Analysis of 
Variance on Grades 
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Although this analysis is quite similar to those presented in the preceeding two appendices, 
there is one structural change. The sampling periods were not randomly selected by the 
experimenter but were fixed. The variable (Op) in the table generaged by the Cornfield-Tukey 
Algorithm is therefore not 1 as in the previous designs but is zero (0), The relevant values are 
presented in Table L 



TABLE 1 

Matrix for Determining Expected 
Mean Squares for Grades Data 



Effect 


1 


k 


i 


i 


1- ori 


0 


q 


r 


n 


2. 


P 


0 


r 


n 


3. 7j 


P 


q 


0 


n 


4- ^i(jk) 


P 


1 


1 


1 


5. a^kl 


0 


0 


r 


n 


6. aTji 


0 


q 


0 


n 


7. ^7kj 


P 


0 


0 


n 


8. a^ii (kj) 


0 


1 


1 


1 


9- a/^Tjkl 


0 


0 


0 


n 
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Only the expected mean squares that are dilTerent than those given in Appendix C are listed 
below. 

E(MSc) = p5- + pqn 5- 
E(MSs) = p6^ 
H(MSBc) = pn6^^ 

The appropriate F ratios for these effeets now become: 
MaiiiEtTectB: F = MS(B) / ;\1S(S) 
Main HtTect C: F = MS(C) / MS(S) 

The components of variance information is included here in Table 2 and the F ratios needed 
for testing the significance of effects on the Grade data is in Table 3. 

TABLE 2 



Effects 



Analysis of Variance Table for Grades Data 

Degrees of 

Sum of Squares Freedom Mean Square 



Among Students 


69.64022 


299 




Schools (B) 


14.13480 


5 


2.82696 


Sex (C) 


1.23035 


1 


1.23035 


B X C 


0.68282 


5 


0.13656 


Students (S) 


53.59225 


288 


0.18608 


Within Students 


15.28232 


300 




Grades (A) 


1.42984 


1 


1.42984 


A X B 


0.37385 


5 


0.07477 


Ax C 


0.27864 


1 


0.27864 


A X B X C 


0.21478 


5 


0.04396 


Ax S 


12.98521 


288 


0.04509 



TOTAL 



84.92254 599 
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TABLE 3 

F Ratios for Testing 
Significance of Effects on Grades Data 

Degrees Value Approximate 
of of F Tabled 

Effect F Ratio Freedom Ratio F Value {a = .05) 

Marking 



Periods (A) 


MS (A) / MS (AS) 


1/288 


31.71* 


3.86 


Schools (B) 


MS(B)/MS(S) 


5/288 


15.19* 


2.26 


Sex (C) 


MS (C) / MS (S) 


1/288 


6.61* 


3.86 


Students (S) 










AxB 


MS (AB) / MS (AS) 


5/288 


1.66 


2.26 


AxC 


MS (AC) / MS (AS) 


1/288 


6.18* 


3.86 



B X C 

; 

/^Significant at better than the specified .05 level. 
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APPENDIX F 
Standardized Test Results 
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